Add example for a job writing to a Unity Catalog volume #51
Conversation
| @@ -0,0 +1,60 @@ | |||
| { | |||
There was a problem hiding this comment.
This should render fine in the Github repo files. For example: https://github.com/databricks/cli/blob/main/internal/testdata/notebooks/py1.ipynb
shreyas-goenka
requested review from
andrewnester,
denik,
lennartkats-db and
pietern
pietern
changed the title
pietern
changed the title
knowledge_base/save_job_result_to_volume/resources/my_volume.volume.yml
Outdated
Show resolved
Hide resolved
knowledge_base/save_job_result_to_volume/resources/top_ten_trips.job.yml
Outdated
Show resolved
Hide resolved
knowledge_base/save_job_result_to_volume/resources/top_ten_trips.job.yml
Outdated
Show resolved
Hide resolved
| name: my_volume | ||
| # We use the ${resources.schemas...} interpolation syntax to force the creation | ||
| # of the schema before the volume. Usage of the ${resources.schemas...} syntax | ||
| # allows Databricks Asset Bundles to form a dependency graph between resources. |
There was a problem hiding this comment.
Do we need to go into this here?
There was a problem hiding this comment.
We have had multiple SAs reach out to us and ask how to sequence the resource creation. Given that folks will often use schemas with their volumes it feels relevant to keep this bit here.
There was a problem hiding this comment.
Thanks. Linking the warning PR for posterity: databricks/cli#1989
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "file_path = dbutils.widgets.get(\"file_path\")\n", |
There was a problem hiding this comment.
This doesn't actually work does it? Without a dbutils.widgets.text() and/or widgets section in the ipynb JSON below
There was a problem hiding this comment.
Works fine:
(.venv) ➜ cli git:(detect/schema-dep) databricks fs cat dbfs:/Volumes/main/shreyas_goenka_hello_world/my_volume/hello_world.txt -p dogfood
Hello World!%
knowledge_base/write_from_job_to_volume/resources/hello_world.job.yml
Outdated
Show resolved
Hide resolved
|
Hi @shreyas-goenka and @pietern : Does DAB have a similar capability to the notebook SQL cmd CREAT SCHEAM IF NOT EXIST ? Could you please confirm on this ?, our deployment is failing with the following error [2025-02-04T11:14:03.488Z] Error: cannot create schema: Schema 'xyz_sv_cd_sch' already exists |
|
Application teams has already created these schemas and volumes using a notebook in the Dev environment, but not yet in Test and Prod. so we are trying to observe the behavior in the dev environment using DAB deployment when the Schema and volume already exists. But the deployment failed. |
This example demonstrates how a job can write a file to a Unity Catalog volume.